Multi-sensor sequential calibration system

ABSTRACT

Techniques for performing a sensor calibration using sequential data is disclosed. An example method includes receiving, from a first camera located on a vehicle, a first image comprising at least a portion of a road comprising lane markers, where the first image is obtained by the camera at a first time; obtaining a calculated value of a position of an inertial measurement (IM) device at the first time; obtaining an optimized first extrinsic matrix of the first camera by adjusting a function of a first actual pixel location of a location of a lane marker in the first image and an expected pixel location of the location of the lane marker; and performing autonomous operation of the vehicle using the optimized first extrinsic matrix of the first camera when the vehicle is operated on another road or at another time.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent document is a continuation of and claims priority to U.S.patent application Ser. No. 17/150,271, filed on Jan. 15, 2021. Theaforementioned application is incorporated herein by reference in itsentirety.

TECHNICAL FIELD

This document describes techniques to perform multi-sensor calibrationfor sensors located in a vehicle with pre-defined and/or endurablemarkers.

BACKGROUND

A vehicle may include cameras attached to the vehicle for severalpurposes. For example, cameras may be attached to a roof of the vehiclefor security purposes, for driving aid, or for facilitating autonomousdriving. Cameras mounted on a vehicle can obtain images of one or moreareas surrounding the vehicle. These images can be processed to obtaininformation about the road or about the objects surrounding the vehicle.For example, images obtained by a camera can be analyzed to determinedistances of objects surrounding the autonomous vehicle so that theautonomous vehicle can be safely maneuvered around the objects.

SUMMARY

This patent document describes example multi-sensor sequentialcalibration techniques to calibrate multiple sensors located on or in avehicle.

An example method of calibrating one or more sensors for an autonomousvehicle, comprises receiving, from a first camera located on a vehicle,a first image comprising at least a portion of a road comprising lanemarkers, where the first image is obtained by the camera at a firsttime; obtaining a calculated value of a position of an inertialmeasurement (IM) device at the first time; obtaining an optimized firstextrinsic matrix of the first camera by adjusting a function of a firstactual pixel location of a location of a lane marker in the first imageand an expected pixel location of the location of the lane marker, wherethe expected pixel location is based at least on a previously knownfirst extrinsic matrix of the first camera and the position of the IMdevice; and performing autonomous operation of the vehicle using theoptimized first extrinsic matrix of the first camera when the vehicle isoperated on another road or at another time.

In another example aspect, the above-described methods are embodied inthe form of processor-executable code and stored in a non-transitorycomputer-readable storage medium. The non-transitory computer readablestorage includes code that when executed by a processor, causes theprocessor to implement the methods described in the embodiments.

In yet another example embodiment, a device that is configured oroperable to perform the above-described methods is disclosed.

The above and other aspects and their implementations are described ingreater detail in the drawings, the descriptions, and the claims.

BRIEF DESCRIPTION OF THE DRAWING

FIG. 1 shows a top view of a tractor unit of an autonomous semi-trailertruck comprising sensors for performing the example multi-sensorsequential calibration process.

FIG. 2 shows an example scenario where a vehicle driven on a roadperforms the multi-sensor sequential calibration process.

FIG. 3 shows an example flow diagram of performing multi-sensorsequential calibration.

FIG. 4 shows an example block diagram of a computer that may be locatedin a vehicle to perform an example multi-sensor sequential calibrationprocess.

FIG. 5 shows an example factor graph representation for the multi-sensorsequential calibration process described herein.

FIG. 6 shows an example of two images obtained by two cameras that aresetup for Epipolar geometry.

DETAILED DESCRIPTION

An autonomous vehicle may include sensors such as cameras, Radars, andLight Detection and Ranging (LiDAR) sensors mounted on the autonomousvehicle to obtain sensor data items (e.g., an image from camera, pointcloud map from LiDAR sensor, or radar points from Radar) of one or moreareas surrounding the autonomous vehicle. These sensor data items can beanalyzed by a computer which may be on-board the autonomous vehicle toobtain information about the road (e.g., location of lane markers on theroad) or about the objects surrounding the autonomous vehicle (e.g.,distance to objects located around the autonomous vehicle). However, thesensors on the autonomous vehicle need to be calibrated so that acomputer on-board the autonomous vehicle can precisely or accuratelydetect, for example, locations of the lane markers on the road or alocation of an object in front of the autonomous vehicle.

FIG. 1 shows a top view of a tractor unit of an autonomous semi-trailertruck comprising sensors for performing the example multi-sensorsequential calibration process (also known as multi-sensor sequentialextrinsic calibration technique). For autonomous driving, there aremultiple sensors (e.g., cameras 102 a-102 g) mounted on the vehicle 104at different positions with specified orientation. For example, cameras102 a, 102 c, 102 d may be oriented in a direction facing the front ofthe vehicle 104, camera 102 f may be oriented in a direction facing therear left of the vehicle 104, and camera 102 g may be oriented in adirection facing the rear right of the vehicle 104. The vehicle 104 mayalso include LiDAR sensors 110 a, 110 b located in a front region of thevehicle 104. The vehicle 104 also includes an inertial measurement unit(IMU) sensor 106 in the vehicle 104. Since the IMU sensor 106 mayperform best when the vehicle 104 is moving, the vehicle 104 movesthrough at least some part of the example multi-sensor sequentialcalibration process to calibrate the sensors onboard the vehicle 104 asfurther described in this patent document. The IMU sensor 106 mayprovide measurements (e.g., speed, direction, acceleration, angularvelocity, etc.,) at a pre-determined frequency (e.g., 40 Hz or 50 Hz).

The vehicle 104 includes a global positioning system (GPS) device 108that provides information about a location of the vehicle 104. Since theGPS device 108 and the IMU sensor 106 are installed or located in thevehicle 104, the location of the IMU sensor 106 relative to the GPSdevice 108 is previously known. And, since the GPS device 108 and themultiple sensors (e.g., cameras 102 a-102 g) are installed or located inthe vehicle 104, the location of each of the multiple sensors relativeto the GPS device 108 is also previously and approximately known. Thus,a location information provided by the GPS device 108 can be used toobtain the approximate locations of the IMU sensor 106 and/or themultiple sensors on the vehicle 104. The approximate locations of theIMU sensor 106 and/or the multiple sensors (e.g., cameras 102 a-102 g)are provided as initial values that can be fine-tuned with the examplemulti-sensor sequential calibration process described herein. The GPSdevice 108 may provide measurements (e.g., location) at a pre-determinedfrequency (e.g., 5 Hz or 10 Hz). In some embodiments, the IMU sensor 106may provide measurements at a frequency that is higher than that of themeasurements provided by the GPS device 108.

The example multi-sensor sequential calibration process described hereincan calibrate multiple sensors at least based on: (1) an associationdetermined between sequential sensor data of a road obtained from themultiple sensors (e.g., at least two cameras 102 a, 102 d located on thevehicle 104) and previously stored map information of the road; (2) IMUsensor's 106 measurements; and (3) GPS device's 108 measurements. Theuse of the IMU sensor 106 can make the example multi-sensor sequentialcalibration process an asynchronous process at least because thesequential sensor data (e.g., images) obtained by the multiple sensors(e.g., cameras 102 a-102 g) may not be synchronized with the measurementdata provided by the IMU sensor 106.

FIG. 2 shows an example scenario where a vehicle driven on a roadperforms the multi-sensor sequential calibration process. The vehicle202 includes multiple sensors such as a plurality of cameras and/or aplurality of LiDAR sensors. A first camera 204 and a second camera 206is shown for ease of description. However, multiple additional cameraand/or multiple LiDAR sensors can be located on or positioned on thevehicle 202 (e.g., as shown in FIG. 1 on vehicle 104) to obtain sensordata items of a calibration road segment 212 towards which the vehicle202 can be driven. In some embodiments, the calibration road segment 212may be located next to or behind the vehicle 202. The examplemulti-sensor sequential calibration process described herein for thefirst camera 204 and second camera 206 can be applied to other sensors(e.g., at least two cameras, at least two LiDAR sensors, at least twoRadars, etc.,) that are located on the vehicle 202 and can obtain sensordata items of the calibration road segment 212. The vehicle 202 can bean autonomous vehicle.

The road 208 includes a calibration road segment 212 that has lanemarkers that can be affixed on either side of the road 208. The lanemarkers include a first set of lane blocks 210 a located on a first sideof the road 208, and a second set of lane blocks 210 b located on asecond side of the road 208 opposite to the first side. In each set oflane blocks, one lane block can be separated from another lane block bya pre-determined distance to form a set of broken lines on the road. Alane block 210 a, 210 b can have a rectangular shape and can have awhite color. As shown in FIG. 2 , the first and second set of laneblocks 210 a, 210 b can be parallel to each other and can extend for alength of 100 meters, 150 meters, or 200 meters to enable a computerlocated in the vehicle 202 to perform the multi-sensor calibrationtechniques described in some embodiments.

The road 208 can be straight (as shown in FIG. 2 ) and can have an eventop surface so that the sensors located on the vehicle 202 can easilyobtain sensor data items of the calibration road segment 212. In someembodiments, the road 208 with a calibration road segment 212 can be abuilt as a private road outside or in a facility where the vehicle 202is parked so that when the vehicle 202 is turned on to drive to adestination, the multiple sensors on the vehicle 202 can be calibratedon the private road. In some other embodiments, the road 208 can be apublic road where the calibration road segment 212 may exclude an areawhere broken lines are absent.

A computer, which may be located in the vehicle 202, may store a mapdatabase (shown as 415 in FIG. 4 ). The map database includes thethree-dimensional (3D) world coordinates of each corner 212 a-212 d ofeach lane block. The computer may include a calibration module (shown as420 in FIG. 4 ) that performs operations related to the examplemulti-sensor sequential calibration process by using measured data fromthe IMU as a constraint between consecutive camera poses as the vehicleis driven between a pre-defined starting location on the road and apre-defined ending location on the road. The calibration module candetermine that the vehicle is at the pre-defined starting and endinglocations by comparing the location of the vehicle (obtained by the GPSdevice) with the two pre-determined locations of the starting and endinglocations.

The calibration module can provide an optimized estimate of a camera-IMUsensor transform, T_(c), which describes extrinsic parameters betweenthe camera and the IMU sensor by using Equation (1) and measurementsobtained by the GPS device. The calibration module obtains a location ofthe vehicle at time t from the GPS device located on the vehicle. Thecalibration module can determine a position (also known as pose) of themultiple sensors (e.g., cameras) located on the vehicle and the IMUsensor located on the vehicle based on the location of the vehicleprovided by the GPS device and previously known distances from the GPSdevice to the IMU sensor and the multiple sensors. For example, if thevehicle included at least one camera and an IMU sensor, the calibrationmodule can obtain both the pose of the camera, T_(t) ^(c) and the poseof the IMU sensor, X_(t) at time t. The camera-IMU sensor transform,T_(c) can be determined by the calibration module by using Equation (1):

T _(c) =X _(t) ⊖T _(t) ^(c)  Equation (1)

In this patent document, the LaTeX symbols ⊕ and ⊖ denote thecomposition and the inverse composition operation for a Lie Group. InEquation (1), the Lie group is SE3. The compose operator for the SE3 Liegroup is the composition of 6D transforms. A 6D transform includes(roll, pitch, yaw) information of the sensor (e.g., IMU sensor orcamera), e.g., R that belongs to a three-dimensional coordinate space

³ and (x, y, z) position information of the sensor (e.g., IMU sensor orcamera), e.g., T in a three-dimensional coordinate space

³. The composition of two transforms, A=R^(a), T^(a), B=R^(b),T^(b) inSE3 for C=A⊕B is R^(c)=R^(a)*R^(b), and T^(c)=R^(a)T^(b)+T^(a). Theinverse composition reverses a composition. Equation (1) only describesIMU sensor pose X_(t) and camera pose T_(t) ^(c). Although the IMUsensor pose, X_(t), and the camera pose, T_(t) ^(c), change over time,the camera-IMU sensor transform, T_(c), between them can remainconsistent. The camera-IMU sensor transform, T_(c), belongs to a SE(3)matrix, where SE(3) is a Lie group.

FIG. 5 shows an example factor graph representation for the multi-sensorsequential calibration process described herein. The optimizationtechnique in FIG. 5 is performed by the calibration module. The factorgraph is a model used to represent the optimization problem. Thecalibration module applies optimization to the factor graph to solve thevalue of the variables shown in circles in FIG. 5 in order to minimizeall the loss that is defined by the factors shown in squares ortriangles. The columns shown in FIG. 5 (shown with subscripts 0 to n−1and n in the circles) indicate the variables that are solved by thefactor graph for times 0 to n. The section headings in this patentdocument are meant to facilitate an ease of understanding. Accordingly,one or more features from one section can be combined with one or morefeatures of another section.

I. Variables of the Factor Graph

The variable X_(t) indicates IMU sensor pose at time t, where X_(t)belongs to a SE(3) matrix, where SE(3) is a Lie group. The calibrationmodule can determine the IMU sensor pose X_(t) based on locationinformation provided by the GPS at time t and based on a previouslyknown location of the IMU relative to the GPS device. The variable V_(t)indicates the velocity of the vehicle at time t, where V_(t) belongs toa three-dimensional coordinate space

³. The variable B_(t) indicates an IMU accelerometer (or accel)+gyrobias at time t, where B_(t) belongs to a six-dimensional coordinatespace

⁶. The variable ω_(t) indicates an angular velocity of the vehicle attime t, where ω_(t) belongs to a three-dimensional coordinate space

³. The variable T_(c) is camera-IMU sensor transform or camera extrinsicparameters of a camera, where T_(c) belongs to a SE(3) matrix. Thevariable K_(c) is an intrinsic matrix of the camera, where the intrinsicmatrix includes parameters that describe optical characteristics (e.g.,focal length, distortion, optical center, etc.,) of the camera. Thevariables T_(c) and K_(c) can be separately determined for each of themultiple cameras located on the vehicle.

II. Information Provided for the Factor Graph

As indicated by the legend in FIG. 5 , the squares located on top of thefactor graph are location information provided by the GPS device to thecalibration module at time 1 to time n. The top two squares located toleft of the left-most column are location information provided by theGPS to the calibration module at time 0. FIG. 5 also shows that thebottom two squares located to left of the left-most column are identityprior information that are obtained by the calibration module prior totime 0. An identity prior can be a constraint for which the values, B₀,ω₀, are zero vectors so that the identity for a

³ vector, ω₀, can be (0,0,0), and the identity for the

⁶ vector, B₀, can be (0,0,0,0,0,0). The values for B₀ and ω₀ can enablethe optimization to be fully constrained.

III. IMU Factor for the Factor Graph

FIG. 5 shows an IMU factor that is obtained by the calibration modulebased the GPS measurements and based on IMU accelerometer and gyroscopemeasurements between the states at time X_(t-1), V_(t-1), and X_(t),V_(t), B_(t). For example, as indicated by the IMU factor triangles inFIG. 5 , the calibration module obtains from the GPS device measureddata with which the calibration module can determine a pose X_(t) of theIMU sensor, and the calibration module can obtain from the IMU sensor avelocity V_(t) and a bias value B_(t). The IMU bias, B_(t), can also bepart of the optimization. The accelerometer and the gyroscope in the IMUcan be susceptible to systematic measurement error. That error consistsof gaussian noise and a bias to the actual acceleration and angularvelocity being measured. The bias can be slowly varying and the IMUsensor can change the bias due to changes in, for example, temperature.The bias can be solved for as part of the optimization to improve theaccuracy of the constraint. The IMU sensor's accelerometer can provideacceleration a_(t) in

³, and gyroscope measurements, ω_(t) in

³ at time t between two times j, k (e.g., j<=t<k) to provide aconstraint on X_(j), V_(j), B_(j) and X_(k), V_(k), B_(k) as furtherdescribed below. The calibration module obtains the measured data fromthe IMU sensor at a pre-determined frequency indicated by the IMU factortriangles labeled 502 a, 502 b, and so on as indicated in FIG. 5 .

In some embodiments, the IMU acceleration measurement, a_(t), relates toan estimate of the true acceleration, â_(t), according to Equation (2).

a _(t) =R _(t)(â _(t) −g)+b(a)_(t) +H(a)_(t)  Equation (2)

where R_(t) describes the rotation of the device at time t, g is the 3Dvector for gravity, b(a)_(t) is the bias of the acceleration, and H(a)_(t) is white noise. The IMU factor constrains the device rotationand the bias to be consistent with the measured acceleration.

In some embodiments, the IMU angular velocity measurement, w_(t),relates to an estimate of the true angular velocity, â_(t), according toEquation (3).

w _(t) =ŵ _(t) +b(g)_(t) +H(g)_(t)  Equation (3)

where b(g)_(t) and H(g)_(t) are the bias and the white noise of thegyroscope. The IMU factor constrains the gyroscope bias to match themeasurement.The IMU factor uses the estimates of the true angular velocity toconstrain the device rotation according to Equation (4).

$\begin{matrix}{R_{j} = {R_{i}{\prod\limits_{k = i}^{j - 1}{{Exp}\left( {{\overset{\hat{}}{w}}_{t}\Delta t} \right)}}}} & {{Equation}(4)}\end{matrix}$

where R_(i), R_(j) represent the orientation of the device at times i,j, Exp (·) is the exponential map function, and Δt is the change intime. The exponential map defines a conversion from a vector in

³ to a

³ ^(3×3) rotation matrix.

The IMU factor uses the estimates of the true acceleration to constrainthe device velocity and position according to Equations (5)-(6).

$\begin{matrix}{v_{j} = {v_{i} + {g\Delta t_{i,j}} + {\sum\limits_{k = i}^{j - 1}{{\hat{a}}_{t}{\Delta t}}}}} & {{Equation}(5)}\end{matrix}$ $\begin{matrix}{p_{j} = {p_{i} + {\sum\limits_{k = i}^{j - 1}{v_{k}\Delta t}} + {\frac{1}{2}g\Delta t_{i,j}^{2}} + {\frac{1}{2}{\sum\limits_{k = i}^{j - 1}{{\hat{a}}_{t}\Delta t^{2}}}}}} & {{Equation}(6)}\end{matrix}$

where v_(i), v_(j) represent the device velocity at times i, j, andp_(i), p_(j), the device positions.

The IMU factor also constraints consecutive IMU bias terms to changeslowly by minimizing the error according to Equation (7).

∥b(g)_(j) −b(g)_(i)∥₂ and ∥b(a)_(j) −b(a)_(i)∥₂  Equation (7)

IV. Gaussian Process Factor for the Factor Graph

Once the calibration module obtains IMU sensor measured data for atleast two consecutive times (e.g., 502 a, 502 b), the calibration modulecan obtain the gaussian process factor. The calibration module obtainsthe gaussian factor, which can be a set of at least three values thatdescribe the pose, velocity, and angular velocity between X_(t-1),V_(t-1), ω_(t-1) and X_(t), V_(t), ω_(t) so that the calibration modulecan obtain the gaussian factor between two states as a measurement.Thus, for example, the calibration module can obtain gaussian factor 504a (at least three values that includes a value between X₁ and X₂, avalue between V₁ and V₂, and a value between ω₁ and ω₂), and thecalibration module can obtain gaussian factor 504 b, and so on asindicated in FIG. 5 . One of the functions of treating the cameratrajectory as a gaussian process in this way is that asynchronous camerameasurements can be incorporated into the factor graph as furtherexplained in below for the 3D-2D factor determined by the calibrationmodule. The sensor data (e.g., images) obtained by cameras on a vehiclemay not be synchronous with the measured data provided by the IMUsensor. Thus, poses, velocities, and angular velocities between statescan be interpolated by the calibration module so that measurements fromasynchronous images can be added to the factor graph for the 3D-2Dfactor.

V. 3D-2D Factor for the Factor Graph

The calibration module obtains an image from each of multiple cameras(e.g., a first camera and a second camera located on the vehicle) attime t=m where the image comprises one or more lane blocks (shown as 210a, 210 b in FIG. 2 ). The calibration module determines a 3D-2D factorwhich is determined as a set of values for T_(c), K_(c) and a set ofvalues (e.g., interpolated values) between X_(j), V_(j), ω_(j) andX_(k), V_(k), ω_(k), respectively, for the image captured at time t=m,where j≤m≤k. The interpolation of X_(t) may require two sets of posevariables, velocity variables, and angular velocity variables. Thus, theinterpolation of X_(t) may include six variables in order to solve forthe pose at the time t between the two states. The calibration modulecan determine an expected pixel locations (p″) of a location (e.g.,corner) of a lane block on the road using an IMU sensor pose X_(t), thatmay be interpolated between X_(j) and X_(k)) using Equation (8) for eachof the multiple cameras:

p″=K _(c) X _(t) T _(c) P  Equation (8)

where K_(c) is the intrinsic camera matrix for a camera that obtainedthe image, X_(t) is IMU sensor pose, T_(c) is the extrinsic matrix ofthe camera that obtained the image, and P is the 3D world coordinates ofthe lane block in the image, where P is the 3D world coordinates of thelocation (e.g., corner) of the lane block obtained by the calibrationmodule from the map database.

The loss function for each 3D-2D factor (also known as localizationfactor) can be determined by subtracting the expected pixel location p″from the actual pixel location p′ using a L2 Euclidian norm as shown inEquation (9):

∥P′−p″∥ ₂=0  Equation (9)

where p′ is the actual pixel location of a location (e.g., corner) ofthe lane block in an image in two-dimensional coordinate space. Thus,the calibration module minimizes the difference between p′ and p″ byadjusting the extrinsic matrix T_(c) of the camera to obtain anoptimized extrinsic matrix T_(c) of the camera.

The multi-sensor sequential extrinsic calibration technique for thecameras can be performed by the calibration module as follows: thevehicle drives on a road (e.g., as shown in FIG. 2 ) collecting sensordata (e.g., camera images) along the way; the collected sensor data isuploaded to a computer (e.g., 400 in FIG. 4 ), which can be accessed bythe calibration module. The calibration module obtains the collectedsensor data and constructs the constraints of the factor graph asdescribed herein, which represents the sequence of constraints from thesensor data captured along the calibration route. The calibration modulecan apply a non-linear optimization to the factor graph of constraints.The calibration module can apply non-linear optimization on theconstraints obtained from the IMU, the GPS, and the results of imageprocess to render the pose transforms, which can solves for values ofthe variables that minimizes the total error to all the constraints. Theresult is applicable to the vehicle that obtained the sensor data. Thesame calibration procedure can be repeated for different vehicles.

VI. 2D-2D Factor for the Factor Graph

In some embodiments, the calibration module can determine whether a2D-2D image-to-image factor can be applied for two cameras that have atleast some overlapping field of views. The identity of the two camerashaving at least some overlapping field of views is previously known andmay be stored in a table on the computer (e.g., 400 in FIG. 4 ), wherethe calibration module can access the stored table. For example, for theexample camera setup in FIG. 1 , the calibration module can apply a2D-2D image-to-image factor to images obtained by cameras 102 a and 102b, or 102 c and 102 d, etc., If the calibration module determines that afirst image obtained by a first camera comprises a lane block that isthe same as the lane block in a second image obtained from the secondcamera, where the calibration module determines that the first cameraand the second camera are pre-determined to have overlapping field ofviews, then the calibration module can obtain a 2D-2D image-to-imagefactor 506. Each lane block may include four map points (e.g., for fourcorners), each of which has an identifier (ID). After the calibrationmodule acquires the data association of 2D pixel locations and 3D worldcoordinates between the lane blocks in an image and the map, thecalibration module uses the map point IDs to identify 2D-2D pixel pairsbetween two images obtained from two cameras for which a 2D-2D factorcan be defined.

A 2D-2D image-to-image factor describes a constraint between cameras forimages captured at the same time, which can enable a 2D-2Dcorrespondences of a same lane block in two images of the two cameras tomatch Epipolar geometry. FIG. 6 shows an example of two images 602, 604obtained by two cameras that are setup for Epipolar geometry. In FIG. 6, X 606 is a location (e.g., corner) of a lane block that is captured bythe two images at pixel locations x 608 and x′ 610 in images 602 and604, respectively. The calibration module can determine the Essentialmatrix E constraint between a pair of corresponding 2d points x 608 andx′ 610 using Equations (10) to (14). The Essential matrix describes arelationship between the location of a pixel in one image and thelocation of another pixel in another image. The 2D-2D factor enables thepixel coordinate to lie on Epipolar lines between the two cameras. This2D-2D factor constraint is useful at least because the map points mayhave some slight errors so that this constraint avoids the use of the 3Dworld coordinates in the map; and because the cameras can be consistentwith one another even if they cannot match the map points exactly.

The calibration module applies a 2D-2D factor constraint between T_(c),K_(c), p_(r) ^(c) and T_(d), K_(d), p_(s) ^(d) using Equations (10) to(14), where p_(r) ^(c) is the measured pixel coordinates of a location(e.g., corner) of a lane block obtained from an image from camera_c andp_(s) ^(d) is the measured pixel coordinates of the same location (e.g.,corner) of the same lane block obtained from an image from camera_d,where T_(c) and T_(d) are the camera-IMU sensor transform or the cameraextrinsic parameters of camera_c and camera_d, respectively, and, whereK_(c) and K_(d) are the intrinsic parameters of camera_c and camera_d,respectively:

ΔP=T _(c) ⁻¹ ⊕T _(d)  Equation (10)

ΔP describes the pose transform in SE3 between the two cameras at posesT_(c) and T_(d).

$\begin{matrix}{E = {\Delta{P.R} \times \left\lbrack \frac{\Delta{P.T}}{❘{\Delta{P.T}}❘} \right\rbrack^{\hat{}}}} & {{Equation}(11)}\end{matrix}$

where R and T are the rotation and translation components of thetransform and the [ ]{circumflex over ( )} operator in Equation (11)denotes the mapping from the unit vector to the skew symmetric matrix.

H ^(r) =K _(c) ⁻¹ ×p _(r) ^(c)  Equation (12)

where H^(r) is a matrix that describes a 3D camera ray from the camerathrough the pixel p_(r) ^(c).

H ^(s) =K _(d) ⁻¹ ×p _(s) ^(d)  Equation (13)

where H^(s) is a matrix that describes a 3D camera ray from the camerathrough the pixel p_(s) ^(d).

H ^(rT) ×E×H ^(s)=0  Equation (14)

Equation (14) defines the Epipolar constraint. Since the extrinsicmatrix T_(c) of the camera_c can be determined using the techniquesdescribed in Section V of this patent document, the calibration modulecan adjust the extrinsic matrix T_(d) of the camera_d such that thematrix multiplication shown in Equation (14) equals zero.

VII. Asynchronous 2D-2D Factor for the Factor Graph

In some embodiments, the calibration module can determine whether anasynchronous 2D-2D image-to-image factor can be applied for two camerasthat are pre-determined to have overlapping or non-overlapping field ofviews. For example, for the example camera setup in FIG. 1 , thecalibration module can apply an asynchronous 2D-2D image-to-image factorto images obtained by cameras 102 a and 102 g, or 102 c and 102 f, etc.,In some embodiments, the calibration module can apply an asynchronous2D-2D image-to-image factor to apply a constraint between a pair ofcameras having overlapping or non-overlapping field of views for imagescaptured at different times. The calibration module can apply theasynchronous 2D-2D factor when the calibration module determines that asame block is included in two images obtained by two cameras at, forexample, two different times. The calibration module can also computethe essential matrix constraint, but because the calibration moduleapplies to asynchronous cameras, the constraint is applied between 16total variables: the same four above (i.e., T_(c), K_(c), T_(d), K_(d))plus four sets of X_(t), V_(t), w_(t). Four sets X_(t), V_(t), w_(t) areused by the calibration module to describe the asynchronous 2D-2Dfactor, where the calibration module uses two sets used to interpolatethe first camera pose and two sets to interpolate the second camerapose. For an image captured by a camera at time t=m, between states t=jand t=k, the IMU sensor pose X_(m) is interpolated from X_(j), V_(j),ω_(j) and X_(k), V_(k), ω_(k). For two such IMU sensor poses X_(m) andX_(n) (for another image from another camera captured at time t=n), theonly change in the constraint is shown in Equation (15):

ΔP=T _(c) ⁻¹⊕(X _(m) ⊖X _(n))⊕T _(d)  Equation (15)

The calibration module can apply ΔP from Equation (15) into Equations(11) to (14) to optimize (or adjust) the extrinsic matrices of camera_cand/or camera_d at the same time by minimizing the error functiondescribed in Equations (11) to (14). In some embodiments, if thecalibration module determines that the extrinsic matrix of camera_c hasbeen optimized (e.g., by using the techniques described in Section V orVI), then the calibration module can minimize the error functiondescribed in Equations (10) to (14) by adjusting the extrinsic matrix ofcamera_d.

VIII. Image-3D Factor for the Factor Graph

In some embodiments, the calibration module can also apply animage-to-3D factor to a camera. The image-to-3D factor replaces an exact2D correspondence to a lane block with an image mask of lane markings.The distance transform between a re-projected point and the mask definesthe error.

In some embodiments, the calibration module can include a general lanemark constraint between a whole image and a map. This technique enablesthe formation of constraints to solid lines, in addition to dashed ones.For each solid lane, the HD-map includes 3D world coordinates at aninterval. For each dashed lane, the 3D world coordinates are thelocations (e.g., corners) of the lane dashes, as previously specified.The calibration module projects a map point of a location of a solidline or dashed line to a region of the image with a lane markingdetected from the image. The calibration module forms a constraint froma map point to an image by first applying image processing to the image.Lane marks are found in the camera image with image processing. Theresult is a binary image that indicates which pixels are lane marks andwhich are not. The calibration module projects the map point accordingto Equation (8), which can fall within a lane mark of the image mask andcan provide an expected pixel location p″ of the lane mark. Given anexpected pixel location p″ as defined in Equation (8), the calibrationmodule determines a nearest pixel (lm) of a lane marker in the imageobtained by the camera. The calibration module applies the constraint asdefined in Equation (16) to adjust the extrinsic matrix of the camera(e.g., using Equation (8)) to minimize the difference between theexpected pixel location and the determined nearest pixel of the lanemarker (lm):

∥p″−lm∥ ₂=0  Equation (16)

FIG. 3 shows an example flow diagram of performing a sensor sequentialcalibration. Operation 302 includes a calibration module receiving, froma first camera located on a vehicle, a first image comprising at least aportion of a road comprising lane markers, where the first image isobtained by the camera at a first time. Operation 304 includes thecalibration module obtaining a calculated value of a position of aninertial measurement (IM) device at the first time. Operation 306includes the calibration module obtaining an optimized first extrinsicmatrix of the first camera by adjusting a function of a first actualpixel location of a location of a lane marker in the first image and anexpected pixel location of the location of the lane marker, where theexpected pixel location is based at least on a previously known firstextrinsic matrix of the first camera and the position of the IM device.Operation 308 includes an autonomous operation module performingautonomous operation of the vehicle using the optimized first extrinsicmatrix of the first camera when the vehicle is operated on another roador at another time.

In some embodiments, the method of FIG. 3 , further comprises receiving,from a second camera located on the vehicle, a second image comprisingat least a second portion of the road, where the second portion at leastpartially overlaps with the portion obtained by the first image receivedfrom the first camera; in response to determining that the second imagecomprises the lane marker that is same as the lane marker in the firstimage: determining, from the second image, a second actual pixellocation of the location of the lane marker; determining, based at leaston the first actual pixel location, a first matrix that describes afirst three-dimensional (3D) camera ray from the first camera to thelocation of the lane marker; determining based at least on the secondactual pixel location, a second matrix that describes a second 3D cameraray from the second camera to the location of the lane marker; andobtaining an optimized second extrinsic matrix of the second camera byadjusting a previously known second extrinsic matrix of the secondcamera based on a matrix multiplication of the first matrix, the secondmatrix, and an essential matrix that describes a relationship of thefirst actual pixel location and the second actual pixel location.

In some embodiments, the previously known second extrinsic matrix of thesecond camera is adjusted so that the matrix multiplication is equal tozero. In some embodiments, the second image is obtained by the secondcamera at the first time. In some embodiments, the essential matrix isbased at least on the optimized first extrinsic matrix and the secondextrinsic matrix. In some embodiments, identities of the first cameraand the second camera that have at least some overlapping field of viewis previously known. In some embodiments, the first matrix for the first3D camera ray is determined by multiplying an inverse of a firstintrinsic matrix of the first camera with the first actual pixellocation, and the second matrix for the second 3D camera ray isdetermined by multiplying an inverse of a second intrinsic matrix of thesecond camera with the second actual pixel location.

In some embodiments, a system for calibration of one or more sensors onor in a vehicle is described, where the system comprising a processorconfigured to perform the operations described at least in FIG. 3 and inthis patent document. In some embodiments, the adjusting the function ofthe first actual pixel location and the expected pixel locationcomprises minimizing a difference between the first actual pixellocation and the expected pixel location to zero by adjusting the firstextrinsic matrix of the first camera. In some embodiments, thecalculated value of the position of the IM device is obtained based atleast on location measurements provided by a global positioning system(GPS) device on the vehicle for a second time and for a third time asthe vehicle is driven on the road, the first time is in between thesecond time and the third time, and the measurement provided by the GPSdevice at the third time is immediately adjacent to that provided by theGPS device at the second time.

In some embodiments, the calculated value of the position of the IMdevice at the first time is based on: a first location provided by theGPS device for the second time, a second location provided by the GPSdevice at the third time, and a previously known location of the IMdevice relative to that of the GPS device. In some embodiments, thecalculated value of the position of the IM device is an interpolatedvalue in between a second position of the IM device at the second timeand a third position of the IM device at the third time, the secondposition of the IM device is based on the second location provided bythe GPS device and the previously known location of the IM devicerelative to that of the GPS device, and the third position of the IMdevice is based on the third location provided by the GPS device and thepreviously known location of the IM device relative to that of the GPSdevice.

In some embodiments, a non-transitory computer readable storage mediumhaving code stored thereon, the code, when executed by a processor,causing the processor to implement a method disclosed at least in FIG. 3and/or in this patent document. In some embodiments, the expected pixellocation is based on a first intrinsic matrix of the first camera, theposition of the IM device, the previously known first extrinsic matrixof the first camera, and three-dimensional (3D) world coordinates of thelocation of the lane marker. In some embodiments, the location of thelane marker includes a corner of the lane marker. In some embodiments,the method of FIG. 3 further comprises receiving, from a third cameralocated on the vehicle, a third image comprising at least a thirdportion of the road, where the third image is obtained by the thirdcamera at a time that is different from the first time; in response todetermining that the third image comprises the lane marker that is sameas the lane marker in the first image: determining, from the thirdimage, a third actual pixel location of the location of the lane marker;determining, based at least on the first actual pixel location, a firstmatrix that describes a first three-dimensional (3D) camera ray from thefirst camera to the location of the lane marker; determining based atleast on the third actual pixel location, a third matrix that describesa third 3D camera ray from the third camera to the location of the lanemarker; and obtaining an optimized third extrinsic matrix of the thirdcamera by adjusting a previously known third extrinsic matrix of thethird camera or by adjusting the optimized first extrinsic matrix of thefirst camera using at least the first matrix and a second matrix.

In some embodiments, the adjusting of the previously known thirdextrinsic matrix of the third camera or the optimized first extrinsicmatrix of the first camera is based on a matrix multiplication of thefirst matrix, the second matrix, and an essential matrix so that thematrix multiplication is equal to zero, and the essential matrixdescribes a relationship of the first actual pixel location and thethird actual pixel location. In some embodiments, the essential matrixis based at least on a second position of the IM device when the thirdimage is obtained by the third camera, the position of the IM devicewhen the first image is obtained by the first camera, the optimizedfirst extrinsic matrix, and the third extrinsic matrix. In someembodiments, the first camera has a field of view that at leastpartially overlaps with that of the third camera.

In some embodiments, the method of FIG. 3 further comprises receiving,from a fourth camera located on the vehicle, a fourth image comprisingat least a fourth portion of the road; determining a fourth actual pixellocation of a first location of a first lane marker in the fourth image;applying an image mask comprising a second lane marker to the fourthimage; determining an expected pixel location of a second location ofthe second lane marker; and obtaining an optimized fourth extrinsicmatrix of the fourth camera by adjusting a previously known fourthextrinsic matrix of the fourth camera so that a difference between thefourth actual pixel location and the expected pixel location of thesecond location of the second lane marker is minimized.

FIG. 4 shows an example block diagram of a computer that may be locatedin a vehicle to perform multi-sensor sequential calibration processesdescribed in some embodiments. The computer 400 includes at least oneprocessor 410 and a memory 405 having instructions stored thereupon. Theinstructions upon execution by the processor 410 configure the computer400 to perform the operations described for the multiple sensors andvarious modules (e.g., calibration module 420 and/or autonomousoperation module 425) as described in FIGS. 1 to 3 and 5 to 6 , and/orthe operations described in the various embodiments.

The autonomous operation module 425 can receive sensor data (e.g.,images, PCD, radar data) of an environment in which the autonomousvehicle is operating, where the sensor data is obtained from themultiple sensors onboard the vehicle. The autonomous operation module425 can instruct the autonomous vehicle to operate on a road (e.g.,determine distance to vehicles in front of the autonomous vehicle, turn,apply brakes) based on the sensor data and based on the sensor dataadjusted using the optimized extrinsic matrix obtained from thecalibration module. In an example implementation, the autonomousoperation module can determine a location of an object located in frontof or next to or behind the autonomous vehicle based on a presence ofthe object on an image obtained by a camera onboard the autonomousvehicle and based on the optimized extrinsic matrix of the camera.

This patent document describes example sequential techniques forperforming extrinsic camera calibration of a multi-sensor equippedvehicle. The calibration module in this patent document can use an IMU,a GPS, and an HD map of 3D lane markers to resolve the pose transformsbetween cameras and the IMU. Image processing techniques can be appliedby the calibration module to register views of map points, views of lanelines, and/or shared views of map points between images obtained bydifferent cameras. Shared views include any image pair of cameras thatshares an overlapping field of view, whether the images were acquired atthe same time or at different times, whether the cameras are of the sametype or not, or whether the lenses have the same focal length or not.Constraints between shared views can maximize the agreement of theextrinsic calibration to Epipolar geometry, which can be better than HDmap-only calibration due to the fact that map points may have errors.Non-linear optimization that may be performed by the calibration moduleon the constraints obtained from the IMU, the GPS, and the results ofimage process can render the pose transforms. The example multi-sensorsequential calibration methods can yield precise extrinsic calibrationof multi-view, standard, wide-angle, and telephoto lens cameras withperception up to 1000 m, for an automotive application.

Some of the embodiments described herein are described in the generalcontext of methods or processes, which may be implemented in oneembodiment by a computer program product, embodied in acomputer-readable medium, including computer-executable instructions,such as program code, executed by computers in networked environments. Acomputer-readable medium may include removable and non-removable storagedevices including, but not limited to, Read Only Memory (ROM), RandomAccess Memory (RAM), compact discs (CDs), digital versatile discs (DVD),etc. Therefore, the computer-readable media can include a non-transitorystorage media. Generally, program modules may include routines,programs, objects, components, data structures, etc. that performparticular tasks or implement particular abstract data types. Computer-or processor-executable instructions, associated data structures, andprogram modules represent examples of program code for executing stepsof the methods disclosed herein. The particular sequence of suchexecutable instructions or associated data structures representsexamples of corresponding acts for implementing the functions describedin such steps or processes.

Some of the disclosed embodiments can be implemented as devices ormodules using hardware circuits, software, or combinations thereof. Forexample, a hardware circuit implementation can include discrete analogand/or digital components that are, for example, integrated as part of aprinted circuit board. Alternatively, or additionally, the disclosedcomponents or modules can be implemented as an Application SpecificIntegrated Circuit (ASIC) and/or as a Field Programmable Gate Array(FPGA) device. Some implementations may additionally or alternativelyinclude a digital signal processor (DSP) that is a specializedmicroprocessor with an architecture optimized for the operational needsof digital signal processing associated with the disclosedfunctionalities of this application. Similarly, the various componentsor sub-components within each module may be implemented in software,hardware or firmware. The connectivity between the modules and/orcomponents within the modules may be provided using any one of theconnectivity methods and media that is known in the art, including, butnot limited to, communications over the Internet, wired, or wirelessnetworks using the appropriate protocols.

While this document contains many specifics, these should not beconstrued as limitations on the scope of an invention that is claimed orof what may be claimed, but rather as descriptions of features specificto particular embodiments. Certain features that are described in thisdocument in the context of separate embodiments can also be implementedin combination in a single embodiment. Conversely, various features thatare described in the context of a single embodiment can also beimplemented in multiple embodiments separately or in any suitablesub-combination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asub-combination or a variation of a sub-combination. Similarly, whileoperations are depicted in the drawings in a particular order, thisshould not be understood as requiring that such operations be performedin the particular order shown or in sequential order, or that allillustrated operations be performed, to achieve desirable results.

Only a few implementations and examples are described and otherimplementations, enhancements and variations can be made based on whatis described and illustrated in this disclosure.

What is claimed is:
 1. A method of calibrating sensors, comprising:obtaining, from a first sensor located on a vehicle, a first sensor datacomprising at least a first portion of a road comprising a lane marker;obtaining, from a second sensor located on the vehicle, a second sensordata comprising at least a second portion of the road; and calibratingthe first sensor and the second sensor based on an association betweenthe first sensor data and the second sensor data, a stored mapinformation of the road, and one or more values based on one or moremeasurements provided by one or more sensors on the vehicle.
 2. Themethod of claim 1, wherein the calibrating the first sensor includesobtaining a first set of parameters of the first sensor, wherein thefirst set of parameters include extrinsic parameters of the firstsensor, and wherein the first set of parameters are obtained byminimizing a difference between a first pixel location of a location ofthe lane marker on the road determined from the first sensor data and afirst expected pixel location of the location of the lane marker on theroad determined from the stored map information of the road.
 3. Themethod of claim 2, wherein the difference between the first pixellocation and the first expected pixel location is minimized by adjustinga previously known third set of parameters of the first sensor.
 4. Themethod of claim 2, wherein the first expected pixel location isdetermined from the stored map information of the road, a pose value ofan inertial measurement (IM) device based on measurements provided bythe IM device and a global positioning system (GPS) device, a matrixthat describes optical characteristics of the first sensor, and apreviously known third set of parameters of the first sensor.
 5. Themethod of claim 1, wherein the calibrating the second sensor includesobtaining a second set of parameters of the second sensor, wherein thesecond set of parameters include extrinsic parameters of the secondsensor, and wherein the second set of parameters are obtained byadjusting a previously known fourth set of parameters of the secondsensor that causes a matrix multiplication of an equation to equal zero,and wherein the equation describes a constraint between the first sensorand the second sensor.
 6. The method of claim 5, wherein the equation isbased on: a first matrix that describes a first ray from the firstsensor to a first pixel location of a location of the lane marker on theroad determined from the first sensor data, a second matrix thatdescribes a second ray from the second sensor to a second pixel locationof the location of the lane marker on the road determined from thesecond sensor data, and a third matrix that describes a relationshipbetween the first pixel location and the second pixel location.
 7. Themethod of claim 6, wherein the first sensor data and the second sensordata are obtained at a same time, and wherein the third matrix is basedon a previously known third set of parameters of the first sensor andthe previously known fourth set of parameters of the second sensor. 8.The method of claim 6, wherein the first sensor data and the secondsensor data are obtained at different times, and wherein the thirdmatrix is based on a previously known third set of parameters of thefirst sensor, the previously known fourth set of parameters of thesecond sensor, a first pose value of an inertial measurement (IM) devicewhen the first sensor data is obtained, and a second pose value of theIM device when the second sensor data is obtained.
 9. The method ofclaim 6, wherein the first matrix is based on intrinsic parameters ofthe first sensor and the first pixel location.
 10. The method of claim6, wherein the second matrix is based on intrinsic parameters of thesecond sensor and the second pixel location.
 11. A system forcalibrating sensors, the system comprising a processor configured to:obtain, from a first sensor located on a vehicle, a first sensor datacomprising at least a first portion of a road comprising a lane marker;obtain, from a second sensor located on the vehicle, a second sensordata comprising at least a second portion of the road; and calibrate thefirst sensor and the second sensor based on an association between thefirst sensor data and the second sensor data, a stored map informationof the road, and one or more values based on one or more measurementsprovided by one or more sensors on the vehicle.
 12. The system of claim11, wherein the obtain the first sensor data, the obtain the secondsensor data, and the calibrate the first sensor and the second sensorare performed while the vehicle is driven on the road.
 13. The system ofclaim 12, wherein the vehicle is driven between a pre-defined startinglocation and a pre-defined ending location on the road.
 14. The systemof claim 11, wherein the first sensor and the second sensor include afirst camera and a second camera, respectively.
 15. The system of claim14, wherein the first camera and the second camera have overlappingfield of views.
 16. A non-transitory computer readable storage mediumhaving code stored thereon, the code, when executed by a processor,causing the processor to implement a method comprising: obtaining, froma first sensor located on a vehicle, a first sensor data comprising atleast a first portion of a road comprising a lane marker; obtaining,from a second sensor located on the vehicle, a second sensor datacomprising at least a second portion of the road; and calibrating thefirst sensor and the second sensor based on an association between thefirst sensor data and the second sensor data, a stored map informationof the road, and one or more values based on one or more measurementsprovided by one or more sensors on the vehicle.
 17. The non-transitorycomputer readable storage medium of claim 16, wherein the lane markerincludes a solid line on the road.
 18. The non-transitory computerreadable storage medium of claim 17, wherein the stored map informationincludes world coordinates of the solid line at an interval.
 19. Thenon-transitory computer readable storage medium of claim 16, wherein thelane marker includes a lane block from a plurality of lane blocks on theroad.
 20. The non-transitory computer readable storage medium of claim19, wherein a first pixel location of a location of the lane marker onthe road determined from the first sensor data, and wherein the firstpixel location of the location of the lane marker includes the firstpixel location of a corner of the lane block.